Skip to content

fix: Preserve int64 precision when rebuilding nested values from strings - #2553

Merged
kodiakhq[bot] merged 1 commit into
mainfrom
fix/nested-int64-precision
Aug 3, 2026
Merged

fix: Preserve int64 precision when rebuilding nested values from strings#2553
kodiakhq[bot] merged 1 commit into
mainfrom
fix/nested-int64-precision

Conversation

@erezrokah

Copy link
Copy Markdown
Member

arrow-go v18.7.0 made array.FromJSON decode int64/uint64 exactly, but AppendValueFromString still routes nested values through float64, so list/struct round-trips silently lose precision (-8717895732742165505 becomes -8717895732742165504) and every destination plugin's write test suite fails. Route nested types through UnmarshalOne with UseNumber instead.

@kodiakhq
kodiakhq Bot merged commit 8599fa0 into main Aug 3, 2026
10 checks passed
@kodiakhq
kodiakhq Bot deleted the fix/nested-int64-precision branch August 3, 2026 16:10
kodiakhq Bot pushed a commit that referenced this pull request Aug 3, 2026
🤖 I have created a release *beep* *boop*
---


## [4.96.0](v4.95.3...v4.96.0) (2026-08-03)


### Features

* Expose error classifier ([#2539](#2539)) ([e042d73](e042d73))


### Bug Fixes

* **deps:** Update aws-sdk-go-v2 monorepo ([#2521](#2521)) ([cc5ce53](cc5ce53))
* **deps:** Update aws-sdk-go-v2 monorepo ([#2524](#2524)) ([d080fbf](d080fbf))
* **deps:** Update aws-sdk-go-v2 monorepo ([#2537](#2537)) ([f5b5033](f5b5033))
* **deps:** Update aws-sdk-go-v2 monorepo ([#2540](#2540)) ([579a722](579a722))
* **deps:** Update aws-sdk-go-v2 monorepo ([#2542](#2542)) ([83ceb77](83ceb77))
* **deps:** Update aws-sdk-go-v2 monorepo ([#2548](#2548)) ([6be63a3](6be63a3))
* **deps:** Update github.com/cloudquery/jsonschema digest to 45e7e20 ([#2535](#2535)) ([acb4b1b](acb4b1b))
* **deps:** Update go module directive to v1.26.4 ([#2528](#2528)) ([66974c9](66974c9))
* **deps:** Update go module directive to v1.26.5 ([#2547](#2547)) ([cb54f5d](cb54f5d))
* **deps:** Update golang.org/x/exp digest to c48552f ([#2527](#2527)) ([4b94b83](4b94b83))
* **deps:** Update module github.com/apache/arrow-go/v18 to v18.7.0 ([#2544](#2544)) ([301cb8c](301cb8c))
* **deps:** Update module github.com/cloudquery/cloudquery-api-go to v1.14.12 ([#2516](#2516)) ([2107281](2107281))
* **deps:** Update module github.com/getsentry/sentry-go to v0.47.0 ([#2529](#2529)) ([94da8e3](94da8e3))
* **deps:** Update module github.com/getsentry/sentry-go to v0.48.0 ([#2549](#2549)) ([133be1f](133be1f))
* **deps:** Update module golang.org/x/text to v0.40.0 ([#2550](#2550)) ([dc3e10e](dc3e10e))
* **deps:** Update module google.golang.org/grpc to v1.82.0 ([#2538](#2538)) ([aee8e3f](aee8e3f))
* **deps:** Update module google.golang.org/grpc to v1.82.1 ([#2543](#2543)) ([7182a82](7182a82))
* **deps:** Update opentelemetry-go monorepo ([#2532](#2532)) ([90f446e](90f446e))
* Preserve int64 precision when rebuilding nested values from strings ([#2553](#2553)) ([8599fa0](8599fa0))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
kodiakhq Bot pushed a commit that referenced this pull request Aug 4, 2026
…2556)

Follow-up to #2553, which fixed the write-test helpers. The `scalar` package has the same problem on the read side: `Struct.Set` and `List.Set` decode JSON into `map[string]any`/`[]any` with the default decoder, so int64/uint64 values beyond float64 precision are rounded before they ever reach a builder.

`{"v":-8717895732742165505}` came back as `{"v":-8717895732742166000}`, and `[-8717895732742165505]` as `[-8717895732742165504]`.

Decode with `UseNumber` and teach `Int`, `Uint` and `Float` to accept `json.Number` (delegating to the existing string parse — `json.Number` is a distinct named type, so the existing `case string` never matched it).

This is what still blocks the postgresql destination in cloudquery/cloudquery#23163, whose read path goes through `scalar.NewScalar`/`Set`.
erezrokah added a commit to cloudquery/cloudquery that referenced this pull request Aug 4, 2026
…23163)

This PR contains the following updates:

| Package | Change |
|---|---|
| [github.com/apache/arrow-go/v18](https://github.com/apache/arrow-go) |
`v18.6.0` → `v18.7.0` |
|
[github.com/cloudquery/plugin-sdk/v4](https://github.com/cloudquery/plugin-sdk)
| `v4.95.3` → `v4.96.1` |
|
[github.com/cloudquery/filetypes/v4](https://github.com/cloudquery/filetypes)
| `v4.7.1` → `v4.7.3` |

arrow-go v18.7.0 made `array.FromJSON` decode int64/uint64 exactly.
Every other JSON path that rebuilds nested values still went through
float64, so round-trips silently rounded (`-8717895732742165505` became
`-8717895732742165504`) and the mismatch broke the write test suite
across all Go destinations.

Fixed upstream:

- plugin-sdk v4.96.0
(cloudquery/plugin-sdk#2553) — shared write-test
helpers
- plugin-sdk v4.96.1
(cloudquery/plugin-sdk#2556) — the `scalar`
package, which postgresql reads through
- filetypes v4.7.3 (cloudquery/filetypes#757) —
file-based destinations

Fixed here, by decoding nested values with `UseNumber`:

- 16 `AppendValueFromString` call sites across 12 destinations
- `UnmarshalOne` decoders in bigquery, elasticsearch, meilisearch,
mongodb
- postgresql's `stripNullsFromMarshalledJson`
- mongodb's struct/JSON write path, converting `json.Number` to the
integer types the BSON encoder accepts, and its read path, restoring
nested uint64 values stored as int64 bits
- elasticsearch and meilisearch document decodes, with numeric builders
taught to accept `json.Number`
- two float64-rounded literals in the transformer/basic test

azblob fails with `AccountIsDisabled`, which also fails on main and is
unrelated to this PR.

Supersedes #23230, whose plugin-sdk bump is included here — it fails on
its own because v4.96.0 requires arrow v18.7.0.

---------

Co-authored-by: cloudquery-ci[bot] <271027272+cloudquery-ci[bot]@users.noreply.github.com>
Co-authored-by: erezrokah <erezrokah@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants